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Abstract — A general lossless joint source-channel coding 
(JSCC) scheme based on linear codes and random interleavers for 
multiple-access channels (MACs) is presented and then analyzed 
in this paper. By the information-spectrum approach and the 
code-spectrum approach, it is shown that a linear code with a 
good joint spectrum can be used to establish limit-approaching 
lossless JSCC schemes for correlated general sources and general 
MACs, where the joint spectrum is a generalization of the input- 
output weight distribution. Some properties of linear codes with 
good joint spectra are investigated. A formula on the "distance" 
property of linear codes with good joint spectra is derived, 
based on which, it is further proved that, the rate of any 
systematic codes with good joint spectra cannot be larger than 
the reciprocal of the corresponding alphabet cardinality, and 
any sparse generator matrices cannot yield linear codes with 
good joint spectra. The problem of designing arbitrary rate 
coding schemes is also discussed. A novel idea called "generalized 
puncturing" is proposed, which makes it possible that one good 
low-rate linear code is enough for the design of coding schemes 
with multiple rates. Finally, various coding problems of MACs 
are reviewed in a unified framework established by the code- 
spectrum approach, under which, criteria and candidates of good 
linear codes in terms of spectrum requirements for such problems 
are clearly presented. 

Index Terms — Code spectrum, correlated sources, informa- 
tion spectrum, linear codes, lossless joint source-channel coding 
(JSCC), multiple-access channels (MACs), puncturing. 



I. Introduction 

IN a traditional communication system, source and channel 
coding are treated separately. This ascribes to the celebrated 
separation theorem established by Shannon in 1948, which 
indicates that the separation of source and channel coding 
incurs no loss of optimality when code length goes to infinity. 
While such a separate coding scheme is well motivated for 
the point-to-point case, it can entail significant performance 
losses in more general scenarios. For instance, it is well 
known [ 1 ] that separation between source and channel coding 
is not optimal in the case of transmitting correlated sources 
over a multiple-access channel (MAC), and the interesting toy 
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example in [ 1 ] well illustrates the inferiority of separate coding 
in such cases. Moreover, even for the point-to-point case, it is 
recently reported by Zhong et al. [2] that joint source-channel 
coding (JSCC) usually works more efficiently (in terms of 
the error exponent) than separate coding. Therefore, in some 
applications such as the transmission of correlated sources over 
MACs, it is highly encouraged to adopt a JSCC scheme. 

Analogous to source coding, there are two kinds of problems 
for JSCC, i.e., lossless JSCC and lossy JSCC. In this paper, 
we will only focus on lossless JSCC for MACs (also called 
distributed lossless JSCC). Much work on practical designs 
of lossless JSCC based on linear codes have been done for 
specific correlated sources and MACs, e.g., correlated sources 
over separated noisy channels [3]-[5], correlated sources over 
additive white Gaussian noise (AWGN) MACs [6], [7], and 
correlated sources over Rayleigh fading MACs [8]. However, 
for transmissions of arbitrary correlated sources over arbitrary 
MACs, it is still not clear how to construct a limit-approaching 
lossless JSCC schem^l To explore the answer to this question, 
in this paper, we propose and analyze a general lossless JSCC 
scheme based on linear codes and random interleavers. The 
rest of this paper is organized as follows. 

In Section [III we first introduce and establish required 
concepts and results related to linear codes. A very important 
concept called spectrum is defined by the method of types 
[9], and a series of properties about the spectrum of arbitrary 
mappings and linear codes are given therein. 

Then, in Section |III1 the core problem of designing limit- 
approaching lossless JSCC schemes is investigated. Firstly, 
in Section IIII-BI we show that only pairwise independence 
of codewords is needed in the direct part proof of JSCC, 
where each codeword is randomly generated according to 
a conditional probability distribution of channel input given 
source output, and the conditional probability distribution can 
be optimized to minimize the decoding error probability. Based 
on this observation, we find that a feasible method for lossless 
JSCC is just to find some mechanism for generating a set of 
pairwise independent random codewords subject to a given 
conditional probability distribution. Then, in Section IIII-CI 
by the code-spectrum approach established in Section [III we 
propose and analyze a lossless JSCC scheme based on linear 
codes and random interleavers. It is shown that linear codes 
with good joint spectra can be used to generate codewords 

' Since there is no commonly used definition of performance limit (such as 
capacity) for lossless JSCC problems, the term "limit-approaching scheme" 
generally refers to the scheme that attains the best peiformance predicted by 
theory. 
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satisfying pairwise independence, which then makes it possible 
to estabHsh Hmit-approaching JSCC schemes, provided that a 
good conditional probability distribution is found. 

In Section IIV-AI properties of linear codes with good 
joint spectra are investigated. In particular, a formula on the 
"distance" property of such linear codes is derived, based on 
which, it is further shown that the rate of any systematic codes 
with good joint spectra cannot be larger than the reciprocal 
of the corresponding alphabet cardinality, and that any sparse 
generator matrices cannot yield linear codes with good joint 
spectra. As an interesting further step, in Section ITV-BI design 
of arbitrary rate coding schemes is discussed, and a novel idea 
called "generalized puncturing" is presented therein. 

All the proofs are given in Section [Vl and Section |Vl] 
concludes the paper, in which we offer a complete review of 
various coding problems on MACs in a unified framework es- 
tablished by code-spectrum approach, and summarize criteria 
and candidates of good linear codes for each of the problems 
in terms of spectrum requirements. 

In the sequel, symbols, real variables and deterministic 
mappings are denoted by lowercase letters. Sets and random 
elements are denoted by capital letters, and the empty set 
is denoted by 0. Alphabets are all denoted by script capital 
letters. All logarithms are taken to the natural base e, and 
denoted by In. 

II. Linear Codes, Types, and Spectra 

In this section, we will introduce some new concepts, 
definitions, and notations related to linear codes. An interesting 
approach called "code-spectrum" will be established for the 
design and analysis of (linear) codes. 

Let X and y be two finite (additive) abelian groups, then 
a linear code can be defined by a homomorphism / : A"" — > 
3^™, i.e., a map satisfying 

/(Xi+X2) = /(X1)+/(X2), Vxi,X2eA'" 

where and 3^™ denote the n-fold direct product of X and 
the m-fold direct product of 3^, respectively, and xi and X2 
denote the sequences xi_ia;i^2 • • • xi^n and 0:2,1X2.2 • • • X2.n in 
X"^, respectively. We also define the rate of a linear code / to 
be the ratio n/ra, and denote it by R{f). 

A permutation on n letters is defined to be a bijective map 
a -.Tr, In, where 



T 



{l,2,---,n}. 



We denote the set of all permutations on n letters by Sn and 
the identity permutation in Sn by 1„. Note that, by defining 



a(x) 



i(i)a;o-i(2) • • • ; 



i(n), 



Vx e X " 



any permutation (or interleaver) cr on n letters can be regarded 
as an automorphism on X^\ For each positive integer n, we 
denote by I]„ (or SJ^, S", and so on) a uniformly distributed 
random permutation on n letters, that is 

Pr{I]„ = cr} = e Sn- 

We tacitly assume that different random permutations occur- 
ring in the same expression are independent, and that the 



notations such as E,„ and E„ represent different random 
permutations though it is possible that m — n. 

Throughout the sequel, sequences or vectors are all repre- 
sented by the notations x or {xi)i^x^, and its length is denoted 
by |x|. For any part XiXi+i ■ ■ ■ Xj of the sequence x, we use 
the shortening Xi...j, and for I times repetition of a single 
symbol a, we write a' as the shortening. When we talk about 
linear codes, we always assume that the domain and the range 
are direct products of finite abelian groups. 

Next, we introduce the concept of types in the method of 
types [9]. The type of a sequence x in X^ is the empirical 
distribution Px on X defined by 



1 



Va G X. 



For any distribution P on X, the set of sequences of type P in 
X" is denoted by Tp{X). A distribution P on <Y is called a 
type of sequences in X" if Tp{X) ^ 0. We denote by V{X) 
the set of all distributions on X, and denote hy Pn{X) the set 
of all possible types of sequences in X^. 

Now, we start to define the spectrum, the most important 
concept in this paper. The spectrum of a set A C X^'^ is the 
empirical distribution Sx{A) on Vn{X) defined by 

SAA){P) ^ \{^^Mi^^-P}\ ^ VP G Vn{X) 

where \A\ denotes the cardinality of the set A. Analogously, 
the joint spectrum of a set i? C A:"" x 3^™ is the empirical 
distribution Sxy{B) on Vn{X) x Vm{y) defined by 



Sxy{B){PM) 



|{(x,y)Gi?|Px^P,Py^g}| 
\B\ 



yp ePn{x),Q ePmiy). 

Furthermore, we define the marginal spectra Sx{B) and 
Sy{B) and the conditional spectra Sy\x{B) and Sx\y{B) 
as the marginal distributions and the conditional distributions 
of Sxy{B), respectively, that is 

Sx{B){P)^ V Sxy{B){P,Q) 



Sy{B){Q) 



Sy\x{B){Q\P) 



Sx\y{B){P\Q) ^ 



Qev^iy) 

E 



Sxy{B){P,Q) 

Sxy{B){P,Q) 

Sx{B){P) 
MP satisfying Sx{B){P) ^ 
Sxy{B){P,Q) 

Sy{B){Q) 

VQ satisfying Sy{B){Q) / 0. 



Then naturally, for any given function / : A"" 3^™, we 
may define its joint spectrum Sxy{f), forward conditional 
spectrum Sy\x{f), backward conditional spectrum Sx\y{f), 
and image spectrum Sy(f) as Sxy{T:^{f)), Sy^xiAif)), 
5';f |3;(rl(/)), and Sy{i:\{f)), respectively, where rl(/) is the 
relation defined by {(x, /(x))|x G A""}. Clearly, the forward 
conditional spectrum can be trivially calculated by 

by\x{mQ\p) - sx{x-)iP) ■ 
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If / is a linear code, we may further define its kernel spectrum 
as 5;^ (ker/), where ker/ is the kernel defined by {x|/(x) — 
0™}. We also have 

Sy{.f) = Sy{f{X-)) 

since / is a homomorphism according to the definition of 
hnear codes. 

Careful readers must have noticed that the image spectrum 
and the joint spectrum are virtually the normalized version of 
the "spectrum" in [10] and the generalized and normalized 
version of the input-output weight distribution in [11], respec- 
tively. 

The above definitions can be easily extended to more gen- 
eral cases. For example, we may consider the joint spectrum 
Sxyz{C) of a set C C AT" X 3^'" x Z\ or consider the 
joint spectrum SxiX2yiy2{9) of ^ function g : A""^ x A""^ 

From the definitions given above, we obtain the following 
properties. 

Proposition 2.1: For all P e VniX) and P, e VmiXi) 
(I < i < m), we have 

( n \ 



Sx,-xAllA,){Pu - ■ ■ ,Pm) ^l[SxM'^){Pi. 



i=l 



where 



P) Y{aeAnP{a))\ 



and Ai C A"/'* (1 < i < m). 

Proposition 2.2: Let F^^,^^^ : A"" y"" be the random 
binning function defined in [12], that is, each x G A:"" 
is independently assigned to one of the sequences in 3^™ 
according to a uniform distribution on y™, then we have 

ElSxyiF^^^x-^)] = Sxy{X'' x y"). (D 
Generally, we have 

E[Sxi---x,yi---yt: (^n|^^ \ a',"- )] 
/ fe 

= Sx,...x,y,...yAl[^^'''I[yn- (2) 



i=l 



Proposition 2.3: For any given function / : n'=i "^i"' ^ 
riiLi permutations {ai G <S'„.}-^j and {ct- G 

S,ni}'^=i, we have 

Sxi---Xiyi---yf,{cr' o / o cr) = Sxi---Xiyi---yf,{f), 

where {f o g){x) = f{g{x)) and 

o■'((y^)^eIJ = (CT-(yi))«eife- 

Proposition 2.4: For any given random function F : A"" — + 
3^™, we have 



for any x G X" and y G y™, where 
and 

E[Sxy{F){P,Q)] 



(4) 



a(F)(P,Q) 



i?[5^|;,(F)(Q|P)] 



(5) 



Proposition 2.5: For any given linear code / : A"" y^, 
we have 

Pr{/(0") = y} = l{y - 0™} (6) 

or equivalently 

a(/)(Fo",Py)-|y"l{y = 0'"}. (7) 



If both X and 3^ are the finite field F^, we define a random 
linear code F^^^'^c^ : A"" -> 3^" given by x i — > xA^^^, 
where x represents an n-dimensional row vector, and Anxm 
denotes a random nxm matrix with each entry independently 
taking values in according to a uniform distribution on 
¥q. (Note that for each realization of Anxm, we then obtain 
a corresponding realization of F^,^™, and such a random 
construction has ever been adopted in [13, Section 2.1], [14], 
etc.) Then, we have 



Pr{F^i^^c = y} = q- 



(8) 



Pr{F(x)=y} = |y|-'"a(F)(P.,Fy) 



(3) 



for all X G A'"\{0"} and y G 3"", or equivalently 

a{F^^,^^){P,Q)^l (9) 

for all P G Vn{X)\{Po^} and Q G P™(3^). 

Proposition 2.6: For a given random linear code F : — > 
3^™, we have 

Pr{F(x) = y} = 13^1-" (10) 
Pr{F(x) = y|F(x) = y} 

= |y-™a(F)(Pi_.,Py_y) (11) 

for any unequal x, x G X" and any y, y g 3^™, where 

F(x) = F(x) + y", VxgA"" (12) 

and y'" denotes an independent uniform random vector on 

The proofs of Proposition 12.11 and 12.31 are easy and hence 
omitted in this paper, and the proofs of the other propositions 
are presented in Section IV-AI 

III. Lossless Joint Source-Channel Coding Based 
ON Linear Codes and Random Interleavers 

In this section, a lossless JSCC scheme based on linear 
codes will be presented for the transmission of arbitrary 
correlated sources over arbitrary MACs, and an information- 
spectrum approach ([15, pp. 247-268], [16]) is adopted to 
analyze the performance of the scheme. 
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A. Preliminaries of Information-Spectrum and Formulation of 

the Lossless JSCC problem 

At first, we introduce some concepts of limits in probability, 
which are frequently used in the methods of information 
spectrum [15]. For a sequence {.^„}5^i of real-valued random 
variables, the limit superior in probability and the limit inferior 
in probability of are defined by 



p-lim sup Zn = inf I a 



lim Pr{Z„ > a} = 



and 



p-lim inf Z„ = sup< (3 



lim Pr{Z„ < /3} 



(13) 



(14) 



respectively. 

Secondly, in the methods of information spectrum, a general 
source is defined as an infinite sequence 

x = {x" = (x("\x("\...,x("))}~i 

of n-dimensional random variables X" where each component 

in'] 

random variable (1 < i < n) takes values in the alphabet 
X, and a general channel is defined as an infinite sequence 
W = of conditional probability distributions W"- : 

X" 3^" satisfying 



E 



W^"(y|x) 



Vx e A"" 



for each n = 1,2, Analogously, a i^-tuple of correlated 

general sources are defined to be an infinite sequence 

A 



of random vectors taking values in JlieXjc H"- 

Also, a general multiple-access channel (MAC) with K input 
terminals is defined to be an infinite sequence W = {W""}'^^i 
of conditional probability distributions : Hieift: '^P ~^ 
3^" satisfying 



1, 



for each n= 1,2, To simplify notations, for any nonempty 

set A C Ik, we further define 



-X^A — (-X^i)ieA 

— )ieA 



Next, for any given general sources X and Y, the spectral 
sup-entropy rate of X and the spectral inf-mutual information 
rate between X and Y are defined by 



H{X) = p-lim sup - In ] 



(15) 



and 



/(X; r = p-hmmf - In „ , . (16) 

respectively. Moreover, for any given general sources X, Y, 
and Z, the spectral conditional sup-entropy rate of X given 



Z and the spectral conditional inf-mutual information rate 
between X and Y given Z are defined by 



H{X\Z) = p-lim sup — In , ^ , „ , 



(17) 



and 



7X;VZ =p-limmf-ln ^ (18) 

respectively. 

Finally, based on the notations above, a lossless JSCC 
problem for correlated sources and MACs may be described 
as follows. 

Suppose that a JsT-tuple of correlated general sources 
and an arbitrary general MAC W with K input terminals 
are given, we consider K sequences of separate encoders 
fn) '■ V" {i e Tk) and one sequence of joint decoders 

iei^ V", where Vn may also be expressed 
as {^n^)i^ij^. The whole coding system then works in the 
following way. 

Each encoder <^^'^ {i G Ik) separately encodes the corre- 
sponding n-length source output V^" into a codeword Xf = 
'fin\vi^), and then the codewords X^^ are transmitted over 
the general MAC W"^ simultaneously. The joint decoder 'ipn 
observes the output from the MAC to reproduce the 



estimates V-, 



Tk 



Vn(F") = (#(F")) 



of V, 



To evaluate the performance of such a system, we define the 
decoding error probability e„ as the average decoding error 
probabihty with respect to the probability distribution Py^ 
of the correlated sources, i.e., 

W^"([CHvx.)]1(v'W(vi)).ex.). 

For simplicity, we call a tuple (((^n^ , V'n) of -f^ encoders 
and one decoder with the decoding error probability an 
(n, e„)-code. Then, for a iC-tuple of correlated general sources 
Vr^ and a general MAC W, we define the sources Vx^ being 
e-transmissible over the channel W if there exists an (n, e„)- 
code satisfying 

lim sup e„ < e. 

n— *cx) 

When e = 0, we say that Vi^ is transmissible over channel 
W. 



B. Essential Properties for Designs of JSCC Schemes 

Now, we start to consider the design of a lossless JSCC 
scheme based on Unear codes. Before presenting our scheme, 
we first prove some results to show some essential properties, 
which are helpful for establishing a limit-approaching scheme. 

Lemma 3.1: Let Vjj^ be a i^-tuple of correlated general 
sources and W a MAC with K input terminals. For a given 
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tuple of random encoders, if 

and there exits a family of real functions 
satisfying 

Pr{($W(v,)).eA = iA|(<i>«(v,)).ei. = xx,} 
< g™p(*<-^')(v^,x.) -Q Pr{<l,W(v,) ^ xj (20) 

for all nonempty set A C 2^, any unequal Vi,Vi £ V" (i G 
Zjf), and any Xi,Xi £ A:"" (* £ ^T/f). then there exists a joint 
decoder ^„ such that the decoding error probability 

en < Pi-{{V^^,Xi^,Y") ^ r„.^($„)} + (2^ - l)e-"^ 

(21) 

for all n = 1, 2, . . ., where 7 > and 

Fy.^,(vx,)W^"(y|xx,.) n i^xr|vr(x.|v.) (22) 



r4(<f„) = |(vx,,xx,,y) £ n ^ n ^ y" 



(23) 



«(xA;y|xA<^, v^c) > /i(v^|v^c) 
+ p($(^))(v^,x^)+7 



(25) 



and 



i(xA;y|xAc,VAc) = - In— !^ — 

n Py"|x2^vxJy|xAc,v^c) 

_ 1 ^"(y|xx..) 

/i(va|vaO = - In- ^ ■ -. 



Remark 3.1: At first, note that the requirement (l20b is only 
valid for Pr{(4>l*''(vi))igXA- = xx,} > 0. As for the case 
of Pr{(<I>«(v,)) 

ieiK = xi,} = 0, we need to consider the 
alternative form of ( l20l l. that is 

Pr{(ci>W(vO).6A = ±A, (<i>«(vO).ei, - XX,} 
< gnp(*(,-^')(v^,iA) -Q Pr{$W(v,) ^ x,^} 

Pr{(ci>«(v,)),ex, =xx,}. 



Clearly, for Pr{($i'V»))«eiA- = xx,} = 0, the above 
inequality holds for any p($l'^'')(v^, x^i) < 00. So in the 
sequel, this singular case will be ignored for simplicity, and 
such a simplification has no impact on the correctness of the 
results. Next, for any tuple ($„ )igXA' of encoders, we can 
define 



p(<i>(^))(v^,x^) 



1 Pr{(<l>l'\vO),g^ ^ x^|(C^(vO),gx, ^ XX,} 
aeAPrM^^(vO = xJ 

for all (v^,xa) satisfying Pr{<i>i,''' (vi) — X;} 7^ for each 
i ^ A and 

p($(^))(v^,i^)=0 

for other (v^,xa), so there always exist well-defined func- 
tions {p{^i^^)}A If the tuple {<^^n^)^eIK of en- 
coders are all independent, then the condition (l20b may be 
reduced to the following one: 

Pr{$«(v,)=i,|'i>i^Hv.)=x,} 

< gnp(*(.»))(v„x.) Pr{$W(^,) ^ (26) 

for any unequal Vi,Vi £ V" and any x^jX^ £ A"" (i £ X/^), 
and we have 



max 



p(<i>i-4))(v^,x^) 



P(<i'i'^)(v.,x,) 



(27) 



for any nonempty set A C Ij^. Furthermore, if we define each 
(24) encoder by independently generate each encoder output 



*i'n''(vi) according to a fixed ^a^'^IV" ("l^i), then we have 
Pr{<i>«(v,) - x,|$«(v,) = xj ^ Pr{<i>«(v,) = x,}. 



Therefore 



p($l^)) = 0, ^A^Jk^A^ 



(28) 



(29) 



The above construction has been adopted in the proofs of [15, 
Lemma 3.8.1] and [16, Lemma 1]. We denote this special 
construction by {J-n'')ieik - The significance and implications 
of (|29] l will be discussed later in Remark 13.21 but readers 
should be reminded here that the construction (^n''')ieXfc is 
only one of the methods for constructing encoders satisfying 
(|29] l, since in (|28j, only pairwise independence instead of 
mutual independence are required to yield (|29] l. 

The proof of Lemma 13.11 is presented in Section IV-BI and 
it is an easy extension of the proof of Lemma 1 in [16]. By 
Lemma ITTl we immediately obtain the following theorem. 

Theorem 3.1: Let Vx, be a i^T-tuple of correlated general 
sources, W a general MAC with K input terminals, and 
{^n^)ieXi^ a given tuple of random encoders satisfying (fT9] l 
and ( |20] |. If for a sequence {7n}^i satisfying 

7„ > 0, 7„ — > 0, and n7„ — > 00, as n — > 00 

it holds that 

lim sup Pr { {V^l. , , F") ^ ($„) } < e (30) 
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then there exists an (n, e„) code based on {^n ^)i,zx,^ such that 
Vxj^ is e-transmissible over W, where the probabiHty distri- 
bution of {V^ ,X^^,Y") is defined by and r„,^„($„) 
is defined by y4]l. 

Theorem 13 . 1 1 gives a way for estimating the upper bound of 
the optimal decoding error probabiHty of arbitrary encoding 
schemes for arbitrary sources and MACs. From Theorem 13. II 
a useful corollary follows. 

Corollary 3.1: Let Vj^. be a i^-tuple of correlated general 
sources, W a general MAC with K input terminals, and 
{^n^)i(z2ii a given tuple of random encoders satisfying (fT9] l 
and (|20ll. If it holds that 



- p-limsupp(<i>i^))(VX,X;j) 



(31) 



for all nonempty set A C Z^, then there exists an (n, e„) 
code based on )igXjc such that Vj^ is transmissible over 
W, where the probability distribution of {V£^, Xj^,Y^'-) is 
defined by (l22l i. 

The proof of Corollary 13.11 is presented in Section IV-BI 
Furthermore, assuming that the sources and channels are 
stationary and memoryless, we prove the following corollary. 
For convenience, a i^T-tuple of correlated stationary memory- 
less sources are now represented by a random vector , and 
a stationary memoryless MAC with K input terminals is rep- 
resented by a conditional probability distribution W satisfying 
VF"(yjxxx) = n"=i W{yi\xij^^i), where xx^^i = xi...K,i- 

Corollary 3.2: Let Vxi^ be a X-tuple of correlated station- 
ary memoryless sources, W a stationary memoryless MAC 
with K input terminals, and ($n^)ieix a given tuple of 
random encoders satisfying ( fT9l l and ( |20| |. If it holds that 

n-l 

Pr{$^^(v,) = xj = n Pr{$«K-,^+i...(,+i)^) 

= XijN+i---{j+i)N}, Vi e Ik (32) 



and 



H{Va\Va^) < -IiX2;Y^'\X2.,Vil) 



- - p-lim sup ) {V2'' , Xr ) (33) 

for some positive integer N and all nonempty set A C X^, 

then there exists an (nA^, e„Ar) code based on 

such that Vx,^ is transmissible over W, where the probability 

distributions 

A/" X" y«(vi^.,xi^.,y) = Fyiv (vi^) 

^Y"|x^^(y|xi^c) n ^xf |v;"(x»|v,) (34) 



PxNiv«{xtWt) = Pr{$^^(vj) = xj 



(35) 



and 



FY„N|v„«(xi|vj) = 

n-l 

JJ^ -PxfHA"(2^ijW+l---(j + l)Af l"i,jAf+l---(j+l)Jv)-(36) 

J=0 



Remark 3.2: If the encoders satisfy (|29] |. then 

the condition (l30l l in Theorem 13.11 reduces to the following 
one: 



limsupPr< 



t{X^;YX\X^.,VX. 



^=himsupPrU^iL,^L'^")G U 



(b) 



lim sup Pr { (V^^ , X^^ , y") ^ T„,^„ ($„)} 



< e 



where (a) follows from the definition (I25b and (b) from the 
definition (|24] |. Because the construction { J-n ^)ieXk in Remark 
I3.1l ensures the existence of encoders satisfying ( |29] |. the direct 
theorem in [16, Theorem 1] clearly follows. Analogously, 
when the sources and channels are stationary and memoryless, 
if the encoders i^nl^)ieXK satisfy (l29l l, then we have 



p-iimsupp(<i>i^)(Fr,^r) = o. 

n—>-QO 

Hence by considering all conditional probability distributions 



'(•I-)} 



iV = l,2,. 



and constructing the encoders {J-'nl[)i£Xk according to the 



conditional probability distribution (|36] |. the direct part of 
Theorem 2 in [1] follows explicitly from Corollarv 13.21 The 
above facts tell us that the condition ( |29] l stand out to be 
an important characteristic of good encoders. Furthermore, 
based on the observation at the end of Remark 13.11 we find 
that one feasible approach to construct lossless JSCC is as 
simple as finding some mechanism for generating a set of 
pairwise independent random codewords subject to a good 
conditional probability distribution Px^\v^ of channel input 
given source output. This observation is interesting, since in 
the classical proofs of many theorems in information theory, 
we usually generate a set of independent random codewords. 
However, as we have shown in this paper, only pairwise 
independence is needed in the proof of lossless JSCC problem. 
In case of channel coding, we have known that the pairwise 
independence is sufficient (see [17, Section 6.2] and [18]). 
Now, such a viewpoint is extended to the case of JSCC. 

In light of the above ideas, we define a tuple ($n'')ieijf of 
encoders to be good if it satisfies (T% . ( |20] | and 



sup 

(v^,x^)en,eAvrxn. 



p($i^))(vA,XA) = (37) 



for all nonempty set A C T^. Note that it can be easily shown 
that the inequality 

sup p($1-^))(va,xa) >0 

always holds for any {^n^}iexKy ™d in fact, condition (|37| | 
is equivalent to ( |29] l. In practice, condition (l37T i may be too 
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Strict, SO we define a tuple (<i>n^)ieij^ of encoders to be 5- 
asymptotically good {5 > 0) if it satisfies ( fT9] l. ( |20] l. and 

limsup sup p($^j^^)(va,xa) < (5 

n^oo (v^,x^)en.eAl^."xn.eA'^."- 

(38) 

for all nonempty set A C Tk- We also define a tuple of 
encoders to be 6 -asymptotically good with respect to Vij^ if 
it satisfies (O, (|20|, and 



p-limsupp($i^))(yi,X3) <(5 



(39) 



for all nonempty set A C Ix, where the probability distri- 
bution is define by (|22] |. When 5 = 0, we say the encoders 
are asymptotically good or asymptotically good with respect 
to some given sources. The relations among the above three 
definitions are established in the following proposition. The 
proof is easy and hence omitted. 

Proposition 3.1: Good encoders are asymptotically good, 
and 5-asymptotically good encoders are (5-asymptotically good 
with respect to any given sources. 

C. Lossless JSCC Schemes Based on Linear Codes and Ran- 
dom Interleavers 

Now, let us consider the problem of how to construct 
good encoders. We notice that the random affine function F 
constructed in Proposition 12.61 does exhibit some properties 
of pairwise independence, so it is natural to consider some 
way of constructing good encoders based on F. We obtain 
the following lemma. 

Lemma 3.2: For a given random Unear code Fn : V" 
and a given (quantization) function <?„ : V" x ZY' ^ A""', we 
have 

Pr{$i.„^,„(v) = x} = Fx"MV"(x|v) (40) 
Pr{*F„.,„(v) - x|$F„,g„(v) = x} 
= /3(F„, g„)(v, V, X, x)P^™|v4x|v) (41) 
</3'(K,(?„)(v,x)F;f„|y„(x|v) (42) 



for any unequal v, v G V" and any x, x G X™", where 



P 



*F„,g„(v) = (7„(v,i^„(v)) 

k.^Hv,x)| 



X"^\V 



.(x|v) 



(43) 
(44) 



q-i(v,x) = {ugZ^'|<Z„(v,u) =x} (45) 



/3(-Fn,9n)(v,V,X,x) = 

a(F„)(Pv-v,Pu-u) 



E 



uGij^ "'(v.x), 
uGg^"' (v,x) 



|?;7^(v,x)||g;7i(v,x)| 



(46) 



/3'(F„,9„)(v,x)^ 



E 



Q(F„)(P,Pa,„) 

kn^(v,x) 



(47) 



ueW' ueg„ (v,x) 

The proof of Lemma 13.21 is presented in Section IV-BI 



Remark 3.3: Setting m = n and comparing ( |42] ) with the 
condition ( |20l i or ( |26] l, we immediately have 

P(*F„,gJ = -ln/3'(F„,g„)- (48) 



Recall that in Proposition 12.51 we proved that the random 
linear code F^^'^ satisfies 

a c-RLc (P, Q) = 1 

for all P G P„(V)\{Po4 and Q G P;(W), where V = U = 
Vq. Hence, we have /?'(P,^^,V, g„) = 1 or p{^pn^C qJ = 
0, which means that <i>pRLc „ is a good encoder, that is, it 

can perform as good as the encoders (^ri'^)ieifc- Furthermore, 
according to Remark [TTl we may construct a A'-tuple of good 
encoders satisfying ( |26] l based on a -tuple of independent 
random linear codes, or more specifically, we may only use 
one deterministic linear code K times, as long as it is good 
enough and its alphabet size is greater than or equal to the 
maximum size of the alphabets of all sources. 

According to the relation between a(P„) and p(<&F„,g„) 
established by ( l47T i and ( |48] |. and following the way of defining 
good encoders, we define a random linear code P„ : V" ^ 
to be good if it satisfies 



max aP„ P,Q =1. 



(49) 



It can be shown easily by definition (|5]) and identity ^ that 

a{F,,){P,Q) > 1. 



max 

P£V„iV)\{Pa^}, 



Or else, we may conclude that 

E[Svu{Fn){P:Q)] 



(a) 



E 

P&Vr^iV) 

1 



(b) 1 

< 



|V|" 



E 

PeVr,{V)\{Par^}, 



Pev^vMPo^} 



E[Svu{Fn){P,Q)] 



|V|" 



+ 



E 

^'eP„(V)\{Pon} 



5v(V")(P) 



1 



which contradicts the obvious fact that E[Svu{Fn){P,Q)] is 
a well-defined joint spectrum. Here, (a) follows from (|5]l and 
d?), and (b) from the assumption that 

max a(P„)(P,Q) < 1. 

Pev,AV)\{Po^}, 
QeVi(u) 

Further, if condition ( |49l ) holds, then we have 

a(P„)(P,Q) = l (50) 

for all P G P„(V)\{Pon} and Q G Vi{U). This is because if 
there exists at least one pair (P, Q) (P ^ Pq") such that 

a(P„)(P,Q) < 1 
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or equivalently 

E[Svu{Fn){P, Q)] < SvuiV" X U'){P, Q) 
then by analogous arguments above, we have 
J2 E[Sp,y{F^){P,Q)]<l 

which again contradicts the fact that E[Sxy{Fn){P, Q)] is a 
well-defined joint spectrum. 

We further define a random linear code F„ : V" Z^'" to 
be S-asymptotically good if it satisfies 

limsup max —lna(Fn)(P,Q) < S. (51) 

When (5 = 0, we call it an asymptotically good linear code. A 
good linear code is evidently asymptotically good. 

The relation between good linear codes and good encoders 
are established by the following proposition. The proof is easy 
and hence omitted. 

Proposition 3.2: If the sequence {Fn}^=i of linear codes 
Fn is good, then the sequence {^Fn,qn}?^=i of encoders 
^F„,qn defined by (|43] l is good. If the sequence {Fn}^^i 
of linear codes is (5-asymptotically good, then the sequence 
{^F„,q„}^:=i of encoders is (5-asymptotically good. 

Therefore, Lemma 13.21 provides a feasible scheme for con- 
structing good encoders based on good or asymptotically good 
linear codes, which is depicted by Fig. [T] Since all encoders 
work in the same way in the case of multiple terminals, only 
one encoder is drawn in Fig. [T| for clarity and simplicity. 
For comparison, the linear-codes-based channel coding scheme 
([10], [17]) as well as the linear codes based lossless source 
coding scheme ([19], [20]) are shown in Fig. |2] Note that 
our scheme is in fact the combination of the two schemes 
except that the quantization is now modified to be directly 
correlated with the source output F", which is the most impor- 
tant characteristic of our scheme. It demonstrates the essence 
of cooperation in a network when sources are correlated 
or some side information is available. In practical designs, 
researchers have realized (e.g., [6, p. 992], [7, p. 61]) that 
preserving the correlation among sources in codewords can 
facilitate good cooperations among terminals for simultaneous 
transmission over a MAC, but it is difficult (maybe impossible 
for some cases) to design a linear code which not only has 
good distance (or spectrum) properties but also preserves 
the correlation of sources. In our scheme, such difficulties 
are perfectly eliminated by adding the quantization module 
qn, which is directly correlated with the source output, so 
correlation among sources can be easily preserved in channel 
inputs without disturbing the design of linear codes. Another 
important thing to be noted in this scheme is that, a pair of 
random interleavers is employed in each encoder, an outer 
interleaver E„ and an inner interleaver S;. In general, these 
interleavers are indispensable to the whole system. As for 
stationary memoryless sources, supposing a framework with 
K encoders (thus K pairs of inner and outer interleavers), only 
one out of the K outer interleavers (could be anyone) can be 



omitted. Analogously, only one out of the K inner interleavers 
can be omitted for stationary memoryless MACs. The omitted 
inner and outer interleavers are not required to belong to the 
same pair 

According to the proposed scheme illustrated in Fig.[T] three 
main problems needs to be solved in the design of lossless 
JSCC: 

1) A problem of optimization. How do we find good (or 
at least transmissible) conditional probability distributions 
{Px'^\v^)ieTK for given correlated sources and MACs? 

2) A problem of coding How do we construct a good linear 
code (|49l l or a (5-asymptotically good linear code ( BTT i with 
arbitrarily small 5, and how do we design the quantization 
function qn for a given conditional probability distribution? 

3) A problem of decoding. How do we decode with low 
complexities and permissible loss of optimality? 

Each of the three problems is difficult and deserves further 
investigations. We now discuss some related subjects in the 
following section. 

IV. Further Discussions 

A. Some Properties of Linear Codes with Good Joint Spectra 

According to the definition (|49] | or (ISOl l in Section |lll] and 
the identity ([U in Proposition 12.21 a good random linear code 
F : should satisfy 

E[Sxy{F){P,Q)] = E[Sxy{F^^\x^){Pm 

for all P G Vn{X)\{PQ,^} and Q G r,n{y), which tells us 
that a good (random) linear code has the same joint spectrum 
as the random binning function (when P ^ Pq"). 

About this observation, we would like to give some com- 
ments here. Note that the definition (|49] l may be rewritten as 

Sxy{Fn){PM) 



max 

Qev^iy) 



E 



Sxy{X^xy--){P,Q) 



= 1. 



However, it is not equivalent to the following equality 

Sxy{Fn)iP,Q) 



E 



max 



PeT'„(A-)\{Po"}, SxyiX"" X 3^™)(P,Q) 
Qev^iy) 



= 1. 



So we can not claim that there exists a single good linear code 
/ in the sense of f4% even though there exists a good random 
linear code F (e.g., F^^'^)- To further explore the relation 
between a good random linear code and its sample codes, we 
prove the following proposition. 

Proposition 4.1: Let F„ : X" ym^ i^g ^ random linear 
code whose rate satisfies sup„>i -R(F„) < oo. If F„ is good, 
there exists at least one sample code /„ such that 



max 



Sxy{fn){P.Q) 



p^v„(x)\{Po^}, Sxy{X^ X 3^™")(P,Q) 
< {n + {rrin + 



(52) 



where ci and C2 are arbitrary constants satisfying ci > \X\ 
and C2 > which implies that /„ is asymptotically good. 

^This problem has been solved partly in [21] very recently. 
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Fig. 1 . The proposed lossless joint source-channel encoding scheme based on linear codes for multiple-access channels 
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Fig. 2. The channel encoding scheme and the lossless source encoding scheme based on linear codes for (multiple-access) channels 



Analogously, if F„ is (5-asymptotically good, then there exists 

at least one sample code /„ which is (5-asymptotically good. 
The proof of Proposition 14.11 is presented in Section IV-CI 
Next, to have deeper understanding of good linear codes, 

we establish the following theorem on the "distance" property 

of good linear codes. 

Theorem 4.1: For a given sequence F = of 6- 

asymptotically good linear codes F„ : 3^™", we have 



lim E[\B{Fn,hx,hy)\]=0 

n — >oc 

for any real numbers hx and hy satisfying 

(hx + S)R{F) +hy < In 13^1 



(53) 



where 



and 



B{Fn,hx,hy) = {{^,y) e Tl{Fn)\ 

^^0",HiP^)<hx,HiFy)<hy} 

R{F) = limsupi?(F„) 



(55) 
(56) 



is called the sup-rate of the code sequence F (we assume that 
R{F) < oo). Furthermore, we have 

p-liminf [(In \X\ + 6)R{F) + min H{Pf,^^^))] 

n — >oo x^Q^ 

> p-liminf min + 5)R{F) + H{Pf„(^))] 

> In 13^1 . (57) 



The proof of Theorem 14.11 is presented in Section IV-CI 
Remark 4.1: The quantity H{Py^) in Theorem 14.11 is just 
the well-known Shannon entropy defined by 

i/(P)^^P(a) In. ^ 

aex 



P{a) 



and here it is used as a substitute for Hamming weight in the 
binary case. Unlike Hamming weight, entropy is not a norm, 
but it does possess the essential properties that we need to 



describe a code. From the second inequality of dSTj i, we find 
that, for asymptotically good linear codes (5 — 0), all the 
input sequences with low entropy except the all-zero sequence 
are mapped into output sequences with high entropy. Next, 
\e.t X and F„ be the random linear code F]^}f^^ 

defined in Proposition 12.51 then it follows from (|57] l that 

Let y™" be the codeword satisfying 



p-liminf min HiPpBLc 



(54) 

and define 



(0) = nmx PpRLc^ (x)(0) 



A„ = 1-Py,.„ (0). 



Clearly, it is just the normalized distance of the code F^^*^^. 
Then we have 



p-liminf [/i(A„) 

n — >oo 

— p-liminf [ft, (Pyn 

n — >oo 

(a) 

> p-liminfi7(Py™„) 



f A„ ln(q- 1)] 
,(0)) + (l-Py. 



,(0))ln(9-l)] 



> p-liminf min H{PpBi.c (^^^) 

n— »00 X^O" q.n.mn 



> (1 



(58) 



where h{x) — — a;lna: — (1 — .t) ln(l — .t) and (a) follows from 
the simple inequality 

H{P) < h{P{0)) + (1 - P(0)) ln{q - 1). 



The inequality iSEi is just the well-known asymptotic Gilbert- 
Varshamov bound (see [22] and [23, Section 2.10.6]). 

By Theorem 14.11 we further prove the following interesting 
results. Their proofs are presented in Section IV-CI 

Corollary 4.1: A code P : A"" — > A"™ (m > n) is said 
to be systematic if all symbols of the input sequence are 
directly included in the output sequence. Then, for any given 
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sequence F — {F„}^]^ of asymptotically good linear codes 
Fn : A"' ^ AT™", we have 



R{F) < 



1 

\X\ 



(59) 



if these codes are systematic. 

Corollary 4.2: Let A" be a finite field, and Anxm be a 
random nxm matrix over X. The average density D{Anxm) 
of the matrix Anxm is defined by 



D{Ar,xm) = £;[Vy l{A„x™(«,j) 7^0}] (60) 

where Anxmihj) denotes the element at the i-th row and 
j-th column in Anxm- Then we conclude that the sequence 
of codes defined by fri(x) = xA„xm„ is not asymptotically 
good if 

liminfi:'(A„xmJ < 1 - 7^- (61) 

n—too \A.\ 

As we can see, when <Y is a finite filed. Corollary 14.11 is 
virtually an easy consequence of Corollary 14.21 and the latter 
establishes the relation between the density of a generator ma- 
trix and its joint spectrum. More interestingly, it demonstrates 
that any sparse generator matrices cannot yield linear codes 
with good joint spectra. This observation shows an important 
difference between the traditional design of channel codes and 
the design of JSCC. In the former case, it is the image of the 
encoding map (the output codewords) that serves as major 
concern, while in the latter case, it is the encoding map itself 
(the generator matrix) rather than its image that determines 
the performance. So the generator matrix deserves careful 
examination in the design of JSCC, even for those good linear 
channel codes in traditional meaning. 

However, it is unnecessary to take a pessimistic attitude 
toward the construction of good linear codes for JSCC. After 
all, only the generator matrix of a linear code is required to be 
dense, and we may still construct good linear codes for JSCC 
based on sparse matrices (e.g., sparse parity-check matrices). 
In our another paper [21], we will investigate the problem of 
constructing linear codes with good joint spectra in detail, and 
an explicit construction based on sparse matrices will be given. 

B. Arbitrary Rate Coding Schemes 

In section Hill we only consider the case when one symbol 
is transmitted per channel use, but the results established 
there can be easily extended to more general cases, including 
correlated sources with different sampling rates and MACs 
with different transmission rates. 

To define correlated sources with different sampling rates, 
we only need to replace the source Vr^^. = {(V'j")iGXK }^i 
by an infinite sequence V^-'''^^ = {(^n*'')iGi/f }^i of random 

(i) I ■ 

vectors and let the alphabet of each Vn be Vj" \ Analogously, 
we may define a MAC with different transmission rates by an 
infinite sequence W = {W"}^^^ of conditional probability 



distribution W" : n.^i,. 



3^" satisfying 



E 



W^"(y"|(x,).eiJ = l 



for all (xi)igi^, e UieiK-'^r"'' ™d each n = 1,2,.... 
Accordingly, the encoder and the decoder are then modified 
to be the maps 0^*'' : V^'" ' — > x^""^ (i £ 2k) and 
ieiK ' respectively. 
Then, for given correlated general sources and gen- 

eral MAC W, we define the sources V^^*") being (i?i^,e)- 
transmissible over the channel W if there exists an (n, e„) 
code satisfying 

lim sup < Ri, Vi G Fk 



71 — ^oo ^n,2 



and 



limsupe,! < e 



where — {Ri)i^2^ denotes a K-tup\e of nonnegative 

real numbers. 

All the results in Section [III] still hold for the new general 
sources and channels, but now we will only concentrate on 
a special case in which /„ ^ = /„ and m„.i — n for all 
i e Fk- In such case, we may define the sources Vj,^ = 
{(^'")ieij< }J5Li being (_R, e)-transmissible over the channel 
W = {W" : UieiK ^ y"'}n=i if there exists an (n, e„) 
code satisfying 

lim sup — < R, lim sup e„ < e. 

n — *oo n — *oo 

In this setting. Lemma [TTI and Theorem |3.1| in Section Hill can 
then be restated as follows by Lemma 14.11 and Theorem 14.21 
respectively. 

Lemma 4.1: Let Vxj^ be a A'- tuple of correlated general 
sources and W a MAC with K input terminals. For a given 
tuple of random encoders, if 

Pr{($«(v,)),ei,- - XI,.} - n Pr{$«(v,0 = xj (62) 

for all e HieiK VY and XI, e Jl ieiji- '^i"' ^'^'^ there 
exits a family of real functions 

satisfying 

Pi-{($«(v,)).e^ = x^|($W(v,0).ei, = xi,,} 
< g«p(*<-))(v,,x.) -Q Pr{coW(v^) ^ (63) 

for all nonempty set A C Fk, any unequal Vi,Vi G V,-" (i G 
Fk), and any Xj,Xi G A"" (i G Fk), then there exists a joint 
decoder "^n such that the decoding error probability 



e„ < Pr{(F^;,Xf,,r") 0r„,,($„)} + (2^ - 1)^ 



(64) 



for all 71 = 1, 2, • • •, where 7 > and 



^y'"X" yJvi,,xi,,y) 



Pv'" (vx.)W^"(y|xi,J n ^^x-iv'^x^lv.) (65) 



ieiK 



P^„l^,„(x,|v,)-Pr{<i>«(v,)=xa 



(66) 
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(67) 



i(xA;y|xA-,VAc) > /i(vA|vyic) 
+ p(*l^^)(vA,x^) + 7 



(68) 



and 



i(xA;y|x^c,v^ 



1 



In- 



w^"(y|xi..) 



71 



/i(va|va<=) = -In- 



n 



Theorem 4.2: Let Vr^. be a if-tuple of correlated general 
sources, W a general MAC with K input terminals, and 
(<I>l*'')igXjf a given tuple of random encoders satisfying 
and ( |63]) . If for a sequence {7n}J^i satisfying 



7„ > 0, 7„ 0, and ?i7„ oo, 



as n ^ cx) 



it holds that 



lim sup — < R 

n — ^oo ^ 



and 



lim sup Pr I (V^l , , ) ^ T,,^„ ($„ ) | 



< e 



then there exists an (n, e„) code based on (<i>l*'')igi^ such 
that Vxj^ is (i?, e)-transmissible over W, where the proba- 



bility disti-ibution of {Vj" , X^^,V) is defined by and 



T'„,-y„($„) is defined by E 

A special case of Theorem 14.21 i.e., the point-to-point 
case, has been presented in [24]. For readers familiar with 
information-spectrum methods, it seems that Theorem 14.21 
brings forth no new groundbreaking results compared with 
Theorem 13. II However, when considering the concrete design 
of encoders with multiple rates under our scheme, some 
interesting results are observed. According to Theorem 14.21 
a feasible approach to lossless JSCC for general rate settings 
is still to generate a set of pairwise independent random code- 
words subject to a good conditional probability distribution 
except that the original conditional probability distribution 
Px^\V" is now replaced by ("^ may not equal n). 

Again by Lemma [J!2l an encoder can be defined by ( |43] ). Note 
that in ( l43T l. if we are given a good linear code F„ : V" — * U' 
with very low rate, that is, — * or //n 3> 1, then for 
most conditional probability distributions Px"|y"(x|v) with 
TO not too large, the only thing that we need to do is simply 
to select an appropriate function q„ satisfying ( l44l i. 

This is an interesting conclusion. In traditional designs of 
channel codes, researchers are usually concerned about how to 
design a code for each rate required by each specific scenario, 
but now, a good low-rate code is enough, and all that we 
need is to design a good quantization function for each rate. 
We call such a method ''generalized puncturing", but there 
is a notable difference between our generalized puncturing 
and the traditional puncturing method. While puncturing is 



traditionally intended for generating a good code with new 
higher rates, the goal of generalized puncturing is to generate a 
good input sequence with multiple rates for a specific channel. 
For some special cases, e.g., the binary symmetric channel, the 
above two goals coincide, but for most cases they do not. 

The generalized puncturing works more like the traditional 
puncturing when the conditional probability distribution satis- 
fies 



which is the usual case for the point-to-point transmission over 
a memoryless channel. Let q : U'^" ^ X he the function 
satisfying 



Px{a) = 



-Ha) 
Mo 



for all a ^ X. Then we may define the encoder by 

3'n(v) = q{ui...i„)q{ui^ + i...2l„) ■ ■ ■q{u(rn-l)lo + l---mlo) 

where v e 1^" and u = F„(v) e Note that I > mlo 
since R{Fn) is assumed to be very small. This mechanism is 
visualized in Fig. [3] Furthermore, if V = U = X = ¥2 and 
Px(0) = Px{^) = 0.5, we may define the function q to be the 
identity function on F2, and then our generalized puncturing 
is almost the same as the traditional puncturing. The only 
difference is that the generalized puncturing operates on the 
randomly interleaved coset codes of F„ (i.e., P„) instead of 
Fn itself. 



V. Proofs 

A. Proofs of Results in Section 1771 

Proof of Proposition \2.2\ By the definition of P^i?|^„ 
we have 



p^.|^RB 



.X^) = y} = \y\ 



for any x e A"" and y G Hence, for any P G Vn{X) 
and any Q G Pm{y), we have 

E[S;,y{TUF^S\;,„)){P,Q)] 
1 



E 



X\ 



^ Pr{F«^|;,„(x)=y} 



\T{i{x)\\r^{y)\ 

This together with Proposition 12.11 concludes ([T]), and the 
identity ^ can be proved analogously. □ 
Proof of Proposition \2.4\ By Proposition 12.31 we have 



E[S;,y{F){PM)]^ E[Sxy{F){PM)] 

for any P G Pn(,X), Q G Pm{y)- Moreover, it is easy to 
show that 

Pr{F(xi) = yi} = Pr{P(x2) = ya} 
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Pandom 
Interleaver 



Linear Code 



Random 
Interleaver 



«1 




Fig. 3. The mechanism of generalized puncturing for each encoder when Px™ | V" C^'" 1^") = Iliii Px{x) 



if = and Py^ = Py^. Therefore, we have 



E[SxyiF)iP^,Py)] 

= E[Sxy{F)iP^,Py)] 
= E[S;,yiTliF)){P^,Py)] 
1 



E 



1 



■ E E 



lX\n E E 



^ l{F{i) = y} 
Pr{F(x) = y} 
Pr{F(x) = y} 



Pr{F(x) - y} 



\yrSxy{X- X 3^™)(Px,Py)Pr{P(x) =y} 



(a) 



where (a) follows from Proposition 12. 11 This together with the 
definition ^ yields Q. □ 
Proof of Proposition \2.5\ The identity ^ is an obvious 
fact in abstract algebra and (|7]i follows easily from ^ and 
Proposition 12.41 

As for the remaining identities, it suffices to show that the 
random linear code F^,^ „ satisfies (HJ, because (|9]l is an easy 
consequence of ([8]l and Proposition 12.41 

Note that both aX and X + Y are uniform random variables 
if X and Y are two independent uniform random variables on 
¥q and a is a nonzero constant in ¥q, and hence dHJ clearly 
follows for all x e A'"\{0"}, y £ J^™. □ 
Proof of Proposition \2.6\ The identity ( fTol i follows easily 
from the definition (fT2l) since F™ is an independent uniform 
random vector on 3^™. To prove (fTTI ). we evaluate 



Pr{P(x 



Pr{P(x)=y,P(i)=y} 

r"=y,F(x-x)=y-y} 
= Pr{rr = F(x),F(i - x) = y - y} 
= 5] P^(/)i?[l{Fr = /(x), /(x - x) = y - y}] 



= 5] P^(/) Pr{Fr = /(x)}l{/(x - x) y - y} 
/ 



where Y{^ = y — F™, (a) follows from the definition (fTSI) . and 
(b) follows from the fact that Y^ is also a uniform random 
vector on 3^™. Then we have 

Pr{P(i) = y|P(x) = y} 
^ Pr{P(x)^y,P(x)=y} 

Pr{P(x) = y} 
= Pr{P(x - x) = y - y} 

'=Vr"«(^^)(^x-x,Py-y) 

where (a) follows from Proposition 12.41 and this concludes 
(El). □ 



B. Proof of Results in Section |^ 

Proof of Lemma \3.1\ In order to define a decoder \I'„ : 

^ n.ei,, we set 

r„,^ (<!>„, VI,,) = |(xx,,,y) ^WK'^ 

(vi^,xi^,y) er„,^($„)l (69) 



r4(<i>„,vx,,) = |(xx,,,y) e n X 3^" 



Clearly 



(vi,.,xi,.,y)er4($„)^ (70) 



(b) 



|3;|-'»Pr{P(x-x)=y-y} 



Suppose that a channel output y e 3^" is received, then 
we define the decoder by vj,, = ipniv) if there exits a 
unique vi,^, G n»eXK '^'^ satisfying ((^l'^(vO)jGi^,y) G 
Pn.7($n, vi^), where ((ysl*'')^^!^ denotes one realization of 
the tuple {^n^)ieii^ of random encoders. If there exists no 
such VI,, or exist more than one such vi, , we define ipn (y) 
to be an arbitrary element in JliGij^ H"- Then, for each sample 
encoder {(pn^)i(EiK generated by ($i*^)igi^, there is a well- 
defined decoder ipn, and we denote by ^„ = {'^n'')ieXK 
the whole random ensemble of the decoders with respect to 
the tuple ($Ti'')iex,, of random encoders. The decoding error 
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probability e„ with respect to the tuple ((<I>l*'')igXA- , ^n) is 
then given by 

= ^V£'^(viA)e„(vi^) (71) 

-x^en.ezA. V." 

where e„(vi^) denotes the decoding error probability of K 
sources' output vj^^ e IlieXA' ^i^' bounded 
above in the following way: 

+ Pr| U {(($«(v,)W.-,y") 

e T„,^($„,VI^.)} 

+ E E Pr{(($W(vO).ex.,y") 



eT„^^($„,vx,)} 



(72) 



where denotes the channel output corresponding to the 
input i<^>'n\^r^)) ieXK- Since the first term on the right-hand 
side of ( |72] | can be written as 

^ Pr{(($«(vO),ex..,r") ^r„,,(<I>„,vx,.)} 

5] w^"(y|xxA) 

Pr{($W(v,)),ex.. -xx^} 

= E w^"(y|xxj n Px^\v^(^^W^) 

E -Pxj^r"|y^^(xxA-,y|vxA-) 
where (a) follows from (fT9] l and (|23T l. it follows that 

E ^yiv(vxA)i?„(vxA) 

- E Pvi^J--^) E 

^i^en^ex,^ (xxA,y)^T„,^(<E.„,vz^) 

y"|y" (xxx^ylvxx) 

Xjf ' Xj^ 

E ^^x" x^.. (vxa , xxa , y) 

(vx,,.,xx,,.,y)^T„,-,(<E.„) 

- Pr{{V^l„Xi^,Y^) ^ T„,^($„)}. (73) 

On the other hand, the second term on the right-hand side of 
(|72| i can be written as 

Cn(vx,.)= E CniAvi^) 
ACIk, 



and 

(Vi7^Vi)ieA 

E E w^"(y|xx.) 

vii^-:VA<^=VAc, (xxA,y)eniexA A-f xy" 

(xxA,y)eT^,^(<f^,vxA) 
Pr{($W(v,)).eXA =xx,|($«(v,)),ex.. =xxa} 

E E w^"(y|xxj 

vi^-:VA<^=VAc, (xxj^,y)GniexA, Xpxy^ 

n Pxr|y,"(x.|v.) E 
'e^^if (XxA-,y)eT^_^(<E'„,vxA), 

Pr{($«(v,)).eA =x^|($«(v,0).ex«- =xx,,} 

vi^-:VAc=VAc, (xxj^,y)GniexA, A-^^xJ^" 



^x? y-iv" (xxA,y|vxA 



E 



(icxA.y)eT^,^(<I.„,vxA), 

XA^=XA<: 



„np(*<.-*')(vA,iA) 



^ E E 

np(*</'')(vA,XA) 



E 



(xxA,y)GT^,-,(<I>„,vxA-), 

XA<:=Xac 



-Px5,y"|y^\(xA-,y|vxK) H ^x,"|y,"(xj|v» 



where (a) follows from (O, (l23]l, and the fact that ^H' (v^) = 
$i*^(v,) foralH G since v^^c = v^^c , and (b) follows from 
(|20] | and (l22T i. Therefore, it follows that 

^ E E E E 

ACXk, VAcen.eAc V," vxa:va-=vac (xAc,y)en,eA- -tTx^" 



E 



np(*k'*')(vA,XA, 



(xxA,,y)eT^^(<I>„,vxA.), 

Xac=Xac 



E ^^x"a (^2:^ 1 y^"^, (xa- , y | vx^ ) 

vAGn.EAvr 

EE E 

AgK, vx^gO^.x^ (ixA-^y)GT„^,^(*„,vxA) 



„np(*(/'')(vA,XA) 



Y[PX^\V^<^^\^^) 
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Pvj^aX^aY" i'^A'^ , x^c , y) for some 7^ > 0. Furthermore, by the definition (l24l l. we have 

<e-"^ E E E lim Pr{(l/£^,XJ^,r")^r„.^(ci>„)} 

ACIk, vx^en.ei,,Vr(Ax^.y)GT„^,(*„,vx^) 
M^"(y|xi;JPyf|yfJvA|vA<=) _[_[ Px^"|y^"(Xi|vi) ""°°ACIx,A#0 



where (a) follows from the fact that Tk is a finite set, and 



<e""" EE EE 

Pv" v"fvT XT v) 7= min 7a > 0. 

= (2^ - l)e-"^, (74) 

Then, from Theorem 13.11 it follows that there exits a code 
where (a) follows from the inequality such that is transmissible over W. □ 



"p(*S,'^')(vA,A^)p /I' - ^ Proof of Corollary\3Ji By regarding every iV consecu- 

^ " -^Y'^iX'^^v^o wl^A" , v^c j jj^g symbols in a sequence as a single symbol, we may define 

< e~"^Pyj|y^.^ (vA|vA<=)W^"(y|xiK) a new source 

implied by ( |25] ). Hence from dTlT ). dT^l l. ( l73b . and ( iTib . it ^ r-^rn loo _ rr^nTVioo 

follows that " ^ ^"=1 " ^ ^"=1 

e„= E and a new channel 

vx^eaex^V." 

< 2^ -Pvr^ (V Jk- ) ( Vl^ ) 

vx^ en.ex^ Furthermore, let 

+ E Pv,-A^lK)Cn{^lK) 

< Pr{(V^^,,X?,,y") ^ T„,^($„)} + (2^ - l)e-"^. 

and 

This completes the proof. □ 
Proof of Corollary 13.71 For all nonempty set A C Ij^, 
we have 

then it follows from (l36b that the triple (1^^- , Xx^ , ^) are 
p-limsup(/i(VJ'|yX<=) ~ ^{^Ai Y"\XAa,VX.) jointly stationary and memoryless. Hence, we have 



Y = 1 = {i^"^}^rLi, 



+ Pi<i>i''MvX,X2)) 



< p-limsup/i(yl|y^.) - p-liminf Vie) 
+ p-limsupp(<i>l-4))(l/i\Xl) 



and 



= ff(VA|VAc)-/(XA;y|XA.,yAe) \Xa^,Va.) 

+ p-\imsup p(^i'^^)(VA ,X'^) r „ ^ ^ -T- ^, „ ■ .c „ .c 

n^oo / V ' ^/ j-Qj- nonempty set A C I^-. The corollary then follows from 

(b) (US and Corollary O □ 

< 

where (a) follows from the properties that 



Proof of Lemma 1X2} From the definition (|43] |. it follows 

that 



p-limsup(X„ + y„) < p-limsupX„ + p-limsupF„ Pr{$i^„^g„ (v) = x} = Pr{P„(v) = u} 

n — >cxD n — 'oo n — *oo 

p-liminf X„ = -p-limsup(-X„), (j^) i^i-i 

n — *oo n — *OQ / ^ ' ' 



(b) 



and (b) follows from the condition dJTT l. Then, by the definition 

of limit superior in probability and dZST l, we obtain that = Px"|y"(x|v) 

P'^i (^lic ' ^lic ' ^" ) ^ r»^^7A where (a) follows from Proposition 122] and (b) follows from 
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the definition ( l44l i. Again from ( |43T l. it follows that 

Pr{*F„,9„ (v) = X, $F„,g„ (v) = X} 

= Pr{F(v) = u,F(v) = u} 



uSg""' (v,x) 



(a) 



J2 \i^r^'a{FnKP^-.,Pu-u) 

uGg""' (v.x) 

(b) Px'"|y"(x|v)Px™|y.(x|v) 



(c) 



|g„\v,x)||g„^(v,x)| 
^ a(F„)(Pv-v,Pu-u) 

uGg~^(v,x), 
ueq""' (v,x) 

/5(-Fn,gn)(v, V,X,x)PxmH/,.(x|v)Px™|y„(x|v) 



where (a) follows from Proposition |2]6] (b) follows from ( l44l i. 
and (c) follows from (|46] l. This concludes the equality (|4T]i, 
and (|42l) is an easy consequence of i4l\ . □ 

C. Proofs of Results in Section WV\ 

Proof of Proposition \4.1\ If _F„ is good, we have 

Sxy{F){P,Q) 



E 



< 1 



for any P e Vn{X)\{Pon} and Q e T'mJJ^)- Then by 
Markov's inequality, we have 

Sxy{F){P,Q) 



Pr 



> (n + l)"i(m„ + l)"^ 



< 



1 



< 



1 



E 



SxyiF){P,Q) 
SxyiX^'xy"'-)iP,Q) 



for all P £ 'Pn{X)\Po^ and Q e Vm^iy)- And furthermore, 
we have 



Pr 



max 



SxyiF){P,Q) 



PeP„(^)\{Po"}, Sxyi^'' X y"")(P,Q) 



> (n + l)=i(TO„ + l)^ 



< 



E 



Pr 



Pe7'„(A')\{Pori}, 



5;,y(j^)(P,Q) 

SxyiX" xy^'^-){P,Q) 



< 

(a) 
< 



\Vn{x)\\r„,Ay)\ 

(n+ 1)^=1 (to„ + 1)^2 
1 



where (a) follows the basic fact |7-'„(A:')| < {n + l)''^' in 
the method of types [9]. This together with the conditions 
ci > \X\ and C2 > |3^| then concludes 



By analogous arguments, we can prove that for any 5- 
asymptotically good random linear code, there exists at least 
one sample code which is (5-asymptotically good. □ 
Proof of Theorem \4.1\ According to the definition ( fSSt . 
we have 



B{F„,hx,h^ 

E 

x:x#0",H(Px)</i;t, 
y:H{Py)<hy 



l{i^„(x)-y} 



E 



Pr{i5^„(x)=y}. 



x:x^O'\H{P^)<hx, 
y:H{Py)<hy 



And for any e > , we further have 

E[\B{F„,hx,hy)\] 

J2 |3^|-™"a(F„)(P.,Py) 



(a) 



x:x^a".H{P^)<hx, 
y:H(Py)<hy 



(b) 
< 



E 



-m„gn(i5+£i) 



x:x#0",if(Px)</i;, 
y:H(Py)<hy 



(c) 

< ^y^-rn„ ^n{5+ei) ^n(hx+e2) ^rn„{hy+e3) 
(d) _ 

< exp{-m„(ln \y\ - (hx + 5)R{F) - hy - e)} (75) 

for sufficiently large n, where (a) follows from Proposition 
12.41 (b) from the definition (ISTl l of a (5-asymptotically good 
linear code, (c) from the inequality 



U 

H(P)<h 



in the method of types [9], and (d) follows from the condition 
( l56l l. Then the upper bound ( fTSl ) together with the condition 
(O concludes the limit (l53T l, because the number e in ( l75b 
can be arbitrarily small. 

Next, for any 7 > 0, we have 

Pr{ min + 5)R{F) + H{Pf,^^^~,)] < In - 7} 

= E[l{mm[{H{P^) + d)R{F) + i/(PF„(x))] 



< E 



<ln|:y|-7}] 

E 



E 



QeVmr^iy) F„{^)<Er^" (y) 



1{{H{P) + 6)R{F) + H{Q) < In \y\ - 7} 

E ^[ E 

1{H{Q) < In \y\ - {H{P) + 5)R{F) - 7} 



(a) 
< 



E 



E 



Pev„(X)\{Por.}, 



\B{F„,H{P), 
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ln\y\-{HiP) + S)RiF)-j)\ 



< (n + 1)1-^1 (m„ + 1)1^1 Gxp{-m„(7 - e)} 
^ 

as n ^ oo, where (a) follows from the definition (l55T l. and (b) 
from (|75ll and the inequality \Vn{X)\ < (n+l)^'^^ concluded 
by the method of types [9]. This together with the definition 
(IT4] | then concludes the second inequality of ( |57b . and the first 
inequality follows easily from the simple inequality i?(Px) < 
hi I A"! for all X G AT". □ 
Proof of Corollary \4.1\ Let a be a nonzero element in 
X. Because {i^ijj^i is asymptotically good, we then have 

p-lim inf H{Pp^(^a")) 

> p-lim inf min [H{P^)R{F) + H{Pf„(^^))] 

>ln|A'| (76) 

by Theorem 14. II If we assume that R{F) > l/\X\, then there 
exists at least a subsequence {i^n, of codes in {Fn}^^i 
such that 

for some positive real number ei and sufficiently large I. Since 
the codes are systematic, we have 

^F„,(a"0(«) > ^+ei (77) 

for sufficiently large /. The entropy H{Pp^^(^a'^i-)) can be 
bounded above by 

/i(Pj.„^(,.,)(a)) + (1 - Pj^„^(a.,)(a))ln(|A'| - 1) 

where h{x) = —a; In a; — (1 — a;)ln(l — x). Define 

gix) = h{x) + il-x) \n{\X\ ~ 1). 

The first- and second-order derivatives of g(x) are then given 
by 

g'{x) = In' 



x{\X\-l) 



and 



9"i^) = - 



x{l — x) 

respectively. We then find that g'{x) < for all < a; < 1, 
which implies g{xi) > g{x2) for ^ < xi < X2 < I- Hence 
we have 

= In I A"! -62 

for sufficiently large /, where €2 — g{jY\) ~ ^^Wi ~^ ^ 
0, and (a) follows from dTTj i and the strictly monotonically 
decreasing property of g. This contradicts the inequality (|76] l 
and hence concludes the corollary. □ 



Proof of Corollary \4.2\ According to the condition 
( I6TI 1. there exits a positive real number ei and a subsequence 
{^1)^1 such that 



< 1 - 



ei 



for sufficiently large /. Furthermore, from the definition ( |60] |. 
it follows that there exits a sequence {h}^^ such that 



—E[Y^ l{A„,x™„,(fc,,j) / 0}] < 1 - ^ 

^ni . . Ki- 



el 



i=i 



for sufficiently large I. Let 1 be the multiplicative identity 
in X, and for each I, define a sequence x; — O'^'^^IO"'^*^'. 
Clearly, we have 

limH(P,,)=0. (78) 
Next, let V^^i = ^^^(x/), and then we find that 

1 

> h ei . 

- \X\ ' 

Hence by similar arguments in the proof of CoroUay 14.11 we 
have 

E[H{Py^^,., )] < H{E[Py^^^, ]) < In \X\ - £3 

for some positive real number £2 and sufficiently large I, where 
(a) follows from the fact that H{P) is a concave function of 
P. This inequality together with (fTsT l obviously contradicts 
Theorem 14.11 with (5 = if we assume that is 
asymptotically good, and hence concludes the corollary. □ 

VI. Conclusion 

In this paper, we propose a lossless JSCC scheme based on 
linear codes and random interleavers for the transmission of 
correlated sources over MACs. To analyze the performance of 
this scheme, we establish an approach called "code spectrum" 
by the method of types. Then we show in Section IIII-CI that 
linear codes with good joint spectra can be used to estab- 
lish limit-approaching coding schemes for arbitrary correlated 
sources and MACs. Furthermore, in Section HV-BI we further 
show that one good low-rate codes can be used to establish 
a limit-approaching coding scheme with multiple rates, and a 
novel idea called "generalized puncturing" is proposed. 

A key concept introduced in this paper is that of "pairwise 
independence of codewords". Its sufficiency for JSCC is in- 
vestigated in detail by Lemma ITTI and Theorem l3.1l in Section 
IIII-BI An important conclusion is thus drawn in Remark 13.21 
that a feasible approach to lossless JSCC is to generate a set 
of pairwise independent random codewords subject to a good 
conditional probability distribution. At the end of Section |III1 
we also list three important problems needed to be solved 
in future, namely, a problem of optimization, a problem of 
coding, and a problem of decoding. Undoubtedly, any essential 
advances on these problems will have vital significance on the 
design of JSCC. 



ACCEPTED BY IEEE TRANSACTIONS ON INFORMATION THEORY (VERSION: MARCH 31, 2009) 



17 



TABLE I 

Criteria and candidates of good linear codes for various coding problems of multiple-access channels 



Coding problems 



Criteria of (5-asymptotically good linear codes 

{Fn -.X" 



Good candidates 



Slepian-Wolf coding (lossless source Kernel spectrum requirements ([19], [20]): 

coding for noiseless MACs, [25]) _ „ 

1 , E[Sx{kcrF„){P)] 

lim sup max — in < o 

n^oo Pev„(x)\{Po^}n Sx{X"){P) 

Image spectrum requirements (implied by [10]): 
lim sup 



Channel coding for MACs (e.g., [27]) 



1 ,^ E[Sy{Fn(X"))m ^ ^ 
max In ^ ^ < o 



Pairty-check matrices of Turbo- 
like codes and LDPC codes (e.g., 
[11], [13], [26]). 



Generator matrices of Turbo-like 
codes and LDPC codes. 



Lossless JSCC for MACs 



Joint spectrum requirements: 
lim sup 



iln. 



E[Sxy(F„)iP,Q)] 



„^oo Pe-p„(A-)\{Po^}, n Sxy{X^ x y"'"){P,Q) 
QePm„(y) 



< 5 



Linear codes with good joint 
spectra can be constructed based 
on sparse matrices. [21] 



Finally, based on our results and previous results [10], 
[19], [20], we can now list all the criteria and candidates of 
good linear codes for various coding problems of IVIACs, and 
all criteria are described in terms of spectrum requirements 
(see Table It is pleasant that we can now answer the 
question "What are good linear codes?" for different coding 
problems in a unified framework established by our code- 
spectrum approach, and note that for the question "What are 
good candidates for JSCC?" we have found the answer, and 
readers may refer to [21] for details. 
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